Augmented reality using eye tracking in a robot assisted srugical system

ABSTRACT

A system augments a surgical procedure by providing information to the user about regions of a surgical site the user is viewing on a displayed endoscopic image. The system uses an eye tracker to determine the region of the displayed image being viewed by the user, and applies computer vision to the image data to identify structures, conditions or disease states within the region. The system provides information to the user, generates alerts, or modifies operational settings for surgical devices based on the identified structures, conditions or disease states.

This application claims the benefit of U.S. Provisional Application No.62/874,976, filed Jul. 16, 2019.

-   Inventor: Stefano Pomati

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to the field of surgical roboticsystems.

BACKGROUND

U.S. Pat. No. 10,251,713, which is owned by the owner of the presentapplication and which is incorporated herein by reference, describes arobotic surgical system that includes an eye tracking system. The eyetracking system detects the direction of the surgeon's gaze and enterscommands to the surgical system based on the detected direction of thegaze.

FIG. 1 is a schematic view of the prior art robotic surgery system 10 ofthe '713 patent. The system 10 comprises at least one robotic arm whichacts under the control of a control console 12 managed by the surgeonwho may be seated at the console. The system shown in FIG. 1 includesmultiple robotic arms 11 a, 11 b, 11 c. Three such arms are shown but alarger or smaller number may be used. Each robotic arm can support andoperate a surgical instrument 9 a, 9 b, 9 c for use on a patient 13. Oneof the instruments is preferably a camera which records the operatingfield inside the patient, while the other instruments may be knownsurgical tools 15, 16.

The arms 11 a, 11 b, 11 c are operated by an electronic control unit 30which causes the arms to perform the movements entered via the console12. The unit 30 will receive the high-level movement commands (forexample, desired position and inclination of the tool supported by therobot) and will execute them, converting them into the correspondingsequences of signals to be sent to the individual motors of the robotarm articulations. Other details of the system 10 are found in the '713patent which is fully incorporated herein by reference.

The console includes input devices 17, 18 which can be gripped by thesurgeon and moved so as to deliver instructions to the system as to thedesired movement and operation of the instruments supported by the arms11 a, 11 b, 11 c.

The surgeon's movements are suitably reproduced by the surgicalinstruments by means of movement of the robotic arms. The input devicesmay be equipped to provide the surgeon with tactile haptic feedback sothat the surgeon can feel on the input devices 17, 18 the forces exertedby the instruments on the patient's tissues.

Each input device will typically operate a robot arm. The '571application describes that where there are two input handles and morethan two arms carrying instruments, the system includes a control on theconsole that allows the surgeon to assign each arm to a desiredinstrument. This allows a surgeon to control of two of the surgicalinstruments disposed at the working site at any given time. To control athird instrument disposed at the working site, one of the two handles17, 18 is operatively disengaged from one of the initial two instrumentsand then operatively paired with the third instrument.

The console may also include a keyboard 19 and/or touch screen and/orother command input devices. These other command devices might include apedal device 20, and a button(s) on or in proximity to one or bothhandles of the input devices 17, 18.

The console 12 has an eye movement tracking system 21 or so-called “eyetracker” for detecting the direction of the surgeon's gaze towards theconsole and for controlling the surgical system depending on the gazedirections detected. In this way, the surgeon may control functions ofthe system by means of movement of his/her eyes.

The tracking system estimates the direction of the surgeon's gazetowards the display 22 and performs selection of the commands associatedwith a zone when it detects a gaze direction which falls within thiszone. In one particular example, the commands associated with selectionareas 29 on the display 22 comprise the commands for assigningparticular ones of the arms to the surgeon input devices. That allowsthe surgeon to alternate control of the robot arms on the two inputdevices without letting go of the input devices, but instead by simplylooking at the corresponding selection areas on the screen. For example,while controlling each of the arms 11 a, 11 c with one of the inputdevices 17, 18, the user might re-assign input device 17 over to arm 11b in order to use or reposition the instrument 9 b within the body. Oncethe task involving movement of instrument 9 b is completed, the surgeoncan rapidly re-assign input device 17 back to robot arm 11 a. Thesesteps can be performed by using the eye tracking features to “drag anddrop” icons on the console display towards icons representing thevarious arms.

In another example described in the '713 patent, the eye tracking systemis used to move the camera based on where the surgeon is looking on thedisplay 22. When this function is enabled (e.g. by entering an inputcommand, such as through pressing of a button on the console, depressinga foot pedal, etc), the movement of the eyes over the image of theoperating field on the screen causes the movement of the robot armsupporting the camera. This can be used to place the zone the surgeonfocused on at the center of the display screen.

The '713 also describes use of the eye tracker to detect the distancebetween the screen and surgeon's eyes as a way to allow the surgeon to“zoom” the camera display in or out. The system enlarges the picture ofthe operating field shown on the screen depending on a variation in thedistance detected. With this feature, the surgeon can intuitivelyperform enlargement of the picture by simply moving his/her face towardsthe screen and, vice versa, increase the viewing area of the operatingfield, thus reducing enlargement, by moving his/her face away from thescreen.

This application described using the eye-tracker of the console (oranother eye tracker) to aid in providing an “augmented reality” to thesurgeon, by helping to display for the surgeon advanced information at alocation on the image display that is most advantageous to the surgeonor most relevant to the type of information being represented.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a robotic surgery system.

FIG. 2 schematically illustrates use of the disclosed system.

FIG. 3 is a functional block diagram showing one embodiment of a methodof using augmented reality with eye tracking.

DETAILED DESCRIPTION

The system will be described in the context of a surgical system of thetype described in the Background and shown in FIG. 1, however it shouldbe understood that the disclosed concepts may be implemented onalternative systems without deviating from the scope of the invention.

An exemplary system includes an eye-tracker positioned at the surgeonconsole. As discussed in the Background, input from an eye-tracker atthe surgeon console allows a processor of the system to determine thepart of the screen at which the user/surgeon is looking. A processor ofthe system is further programmed with a computer vision/image processingalgorithm that causes the processor the analyze real time imagescaptured using a camera disposed in the patient body cavity in which thesurgery is being performed.

The processor is further configured to interact with a database or aknowledge base (e.g. local or cloud based) storing data which may becompared with the real time image data and used to identify certainfeatures, conditions or events within the body cavity. As depicted inFIG. 2, if real time image analysis/recognition is performed and thesystem is connected to a database or a knowledge base (e.g. local orcloud based), that information (gaze of the surgeon and surgical image)can be used to compare the current surgical image and the stored dataand provide some “augmented reality” in the form of an overlay on theimage display in the area at which the surgeon is looking.

More specifically, input from the eye tracker allows determination bythe processor as to where on the displayed image the surgeon is lookingat and where s/he is going to move the surgical instruments carried bythe robotic manipulator. With that information, the processor can causethe following steps to be carried out:

(1) The system recognizes when the surgeon has moved his eyes to a newarea of the screen

(2) Using image data from the camera, the processor responsible forimage processing identifies structures or conditions within the regionof the body cavity that corresponds to the area of the screen beingviewed by the surgeon. For example, the processor may identify bloodvessels, organs, tissue, tissue types, structures or regions to beavoided, tumors or other disease states.

(3) The System takes an action, which may include any of, or multipleones, of the following actions:

a. Alerts

The system generates an alert that alerts the surgeon to the presence oftissue/structures to avoid in the area at which the surgeon is looking.Alerts may include auditory alerts, overlays or graphics, icons or textdisplayed on the image display. Preferable displayed visual alerts arecaused to appear in the region at which the surgeon is looking so thatthe surgeon does not need to turn his/her eyes away from the region ofinterest in order to view them. See the functional block diagram shownat FIG. 3.

b. Information

The system display information to the surgeon relating to the viewedarea, helping him/her understand what s/he is viewing. The informationmight identify the items listed in (2) above using text displayed on theimage display.

c. Active Assistance

Depending on what is identified using the image processing, theprocessor may take an action that facilitates a predicted subsequentsurgical task. For example, it may:

-   -   Enable specific software controls to avoid some movements or        some undesired maneuvers in the detected regions. For example, a        haptic “push” from the user input device may be applied to        resist or prevent the surgeon from moving the surgical        instrument into contact with a structure or tissue to be        avoided, or a scaling factor of the surgical system may be        adjusted to scale down the movement of the surgical instrument        in response to movement of the user input device.    -   If the user input device at the surgeon console is a haptic        interface, modify software parameters to enhance or decrease the        force feedback or the exceeding force thresholds    -   Enable an advanced energy device or set/alter settings on an        electrosurgical unit that will subsequently be used by the        surgeon    -   Automatically activate a fluorescence camera

d. Super Image

Displaying a portion of a pre-operative image (e.g. a CT or MRI scan) ofthe relevant region of the patient's anatomy to provide anenhanced/augmented 2D or 3D image in the spot the surgeon is looking at.For example, the displayed image may be one that is more magnified thanthe real-time camera image, providing the surgeon with a very detailedand “super” image, obtained from the pre-operative images.

All prior patents and applications referred to herein, including forpurposes of priority, are incorporated herein by reference.

What is claimed is:
 1. A system comprising: a camera positionable in abody cavity; a display; an eye tracker; at least one processor and atleast one memory, memory storing instructions executable by the at leastone processor to: cause an image captured by the camera to be displayedin real time on the display; based on input from the eye tracker,determining the region of the displayed image being viewed by the user;identify structures, conditions or disease states from image dataobtained by the camera; obtaining information from a database regardingthe identified structures, conditions or disease states.
 2. The systemaccording to claim 1, where the at least one memory further storinginstructions executed by the processor to perform at least one of thefollowing in response to the obtained information: generate an auditoryalert to the user; generate an overlay relating to the obtainedinformation and display the overlay on the display in the region of thedisplayed image; or enable, or modify operational settings for, asurgical device to be used.
 3. The system according to claim 1, whereinthe surgical device is an energy delivery device, a diagnostic device,or a fluorescence camera.
 4. The system according to claim 1, where thesystem includes a user input device, a robotic manipulator carrying asurgical instrument, the robotic manipulator moveable in response touser manipulation of the user input device, and wherein the at least onememory further storing instructions executed by the processor togenerate or modify haptic feedback generated at the user input devicebased on the obtained information.